Papers with teacher classifier
Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches for transferring supervision across languages require expensive cross-lingual resources. |
| Approach: | They propose a cross-lingual teacher-student method that generates "weak" supervision in a target language using minimal cross-linguistic resources. |
| Outcome: | The proposed method outperforms state-of-the-art methods with a student classifier in 18 languages . it extracts and transfers only the most important task-specific seed words across languages based on translated seed words . |
Intermediate Layer Distillation with the Reused Teacher Classifier: A Study on the Importance of the Classifier of Attention-based Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods underestimate the importance of utilizing the teacher's discriminative classifier and face challenges in establishing proper layer mappings. |
| Approach: | They propose to reuse pre-trained teacher classifiers to improve student performance . they use projectors to match hidden size of the teacher model to student . |
| Outcome: | The proposed method outperforms existing methods on 97.7% of the teacher BERT base without additional trainable parameters. |